The Validity of Sentiment Analysis: Comparing Manual Annotation, Crowd-Coding, Dictionary Approaches, and Machine Learning Algorithms

Sentiment is central to many studies of communication science, from negativity and polarization in political analyzing product reviews social media comments other sub-fields. This study provides an exhaustive comparison sentiment analysis methods, using a validation set Dutch economic headlines compare the performance manual annotation, crowd coding, numerous dictionaries machine learning both traditional deep algorithms. The three main conclusions this article are that: (1) best still attained with trained human or coding; (2) None used come close acceptable levels validity; (3) learning, especially substantially outperforms dictionary-based methods but falls short performance. From these findings, we stress importance always validating automatic text before usage. Moreover, provide recommended step-by-step approach for (automated) projects ensure efficiency validity.